We use semiclassical methods to evaluate the spectral two-point correlation function of quantum chaotic systems with discrete geometrical symmetries. The energy spectra of these systems can be divided into subspectra that are associated to irreducible representations of the corresponding symmetry group. We show that for (spinless) time reversal invariant systems the statistics inside these subspectra depend on the type of irreducible representation. For real representations the spectral statistics agree with those of the Gaussian Orthogonal Ensemble (GOE) of Random Matrix Theory (RMT), whereas complex representations correspond to the Gaussian Unitary Ensemble (GUE). For systems without time reversal invariance all subspectra show GUE statistics. There are no correlations between non-degenerate subspectra. Our techniques generalize recent developments in the semiclassical approach to quantum chaos allowing one to obtain full agreement with the two-point correlation function predicted by RMT, including oscillatory contributions.